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ABSTRACT 



Virtual Learning Environments (VLEs) are learning management 
software systems that synthesize the functionality of computer-mediated 
communications software and online methods of delivering course materials. 
This report focuses on how to evaluate the properties, capabilities and 
orientation of different systems from an educational perspective, and how to 
determine whether the new technology can be embedded into the teaching and 
learning context of an institution. Two different models are explored- -one 
from education, the other from systems modeling- -as a basis for constructing 
a pedagogical evaluation methodology for VLEs. The report concludes that 
among factors slowing the integration of VLEs in higher education 
institutions is the lack of a coherent framework within which to evaluate 
both the pedagogical benefits and the organizational changes required to 
effectively implement it. Another factor includes the extent to which VLEs 
can currently interact with other existing systems. VLEs are inevitably 
designed with a pedagogical model in mind that is not usually made explicit. 
The evaluative strategies in this report are intended to reveal these 
implicit characteristics, to help educators choose a software system that 
reflects the way they wish to teach. An appendix provides a summary of the 
features of several current systems . The Virtual Learning Environments 
Questionnaire is also provided. (AEF) 
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Executive Summary 

Virtual Learning Environments (VLEs) are learning management software systems that 
synthesise the functionality of computer-mediated communications software (e-mail, bulletin 
boards, newsgroups etc) and on-line methods of delivering course materials (e.g. the WWW). 
To date, several different packages have appeared from both leading commercial vendors 
and university-based projects. Other systems are currently under development. 

Most of these systems are intended not simply to reproduce the classroom environment 
-'on-line', but to use the technology to provide learners with new tools to facilitate their 
learning. They aim to accommodate a wider range of learning styles and goals, to encourage 
collaborative and resource-based learning and to allow greater sharing and re-use of 
resources. 

Most of the systems currently available have similar sets of features and range of capabilities. 
However, inevitably some systems are better suited than others to particular educational 
contexts and some are simply better designed than others. Appendix 1 provides a summary of 
the features of a number of current systems. The problems with which this report is concerned 
are twofold. The first is how to evaluate the properties, capabilities and orientation of different 
systems from an educational perspective. The second is how to determine whether the new 
technology can be embedded into the teaching and learning context of a given institution. To 
this end, we explore two different models (one from education, the other from systems 
modelling) as a basis for constructing a pedagogical evaluation methodology for VLEs. The 
educational model was developed and applied to the use of learning technology in higher 
education by Laurillard (1993) as the Conversational Framework. This was, in turn, derived 
from Conversation Theory developed by Gordon Pask. The organisational model is drawn 
from the Viable Systems Model for modelling organisational systems proposed by Stafford 
Beer (1981). The conversational model has been chosen because Laurillard's 1993 book 
"Rethinking University Teaching" has led the way in looking at how Learning Technology 
could be employed to promote more effective and varied teaching styles. We have elected to 
use the organisational systems model because it is essential to understand that when one 
decides to change one element in a system (such as the teaching and learning process by 
introducing new software), it is necessary to consider the impact on other elements of the 
system. It has previously been suggested that this organisational systems approach may be 
applicable in a pedagogical context (Liber, 1998). 

Part two describes the Conversational Framework as developed by Laurillard (1993) and 
constructs an evaluative framework for VLEs based on the model. The use of this framework 
is illustrated with two example VLEs and it is critically examined. In part three the Viable 
Systems Model (VSM) is described and its relevance to the higher education context is 
outlined. A strategy for evaluating VLEs using this framework is then developed and is tested 
with sample VLEs. 

One of the primary differences between the two models is that the conversational model deals 
primarily with the interactions between a single student and a teacher. This form of interaction 
encompasses a good part of the functionality of a VLE, but omits peer group functionality and, 
importantly, tools for allowing the teacher to manage a number of students. These aspects of 
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VLE functionality are better covered by the VSM-based approach. 

The report concludes that amongst the factors that are slowing the uptake of VLEs in Higher 
Education institutions is the lack of a coherent framework within which to evaluate both the 
pedagogical benefits and the organisational changes required to effectively implement it. 
Other factors include the extent to which VLEs can currently inter-operate with other existing 
systems. VLEs are inevitably designed with a pedagogical model in mind, that is usually not 
made explicit. The evaluative strategies developed in this report are intended to tease out 
these implicit characteristics of a system, to help educators choose a software system that 
reflects the way they wish to teach. 

1. Introduction 

1.1 Background 

" we believe that the innovative application of... C&IT holds out much promise for improving 

the quality, flexibility and effectiveness of higher education. The potential benefits will extend 
to, and affect the practice of learning and teaching and research." (Dearing report, 1997, 
13.1). 

Against the background of changes in Higher Education required to achieve the vision of 'a 
learning society' painted by the Dearing Report (1997), this report focuses on recent 
developments in internet / intranet based software systems intended to support teaching and 
learning in Higher Education. In particular we consider several examples of a family of 
systems that are known as virtual learning environments (VLEs). Typically software tools are 
evaluated on the basis of the features they provide, their technical specifications and cost. In 
this report we argue that additional pedagogic criteria are required in order to differentiate 
VLEs with respect to their use in different teaching and learning situations. 

As there are already other comparative reports that evaluate existing VLEs in terms of 
features, technical requirements and cost, this report focuses on developing a theoretical 
basis from which to draw pedagogical evaluation criteria. The aim of this report is not to 
perform a comprehensive evaluation of currently available systems, the fast pace of 
development in this field would quickly make any such attempt obsolete. Instead we have 
opted for the potentially more useful approach of developing an evaluation model based on 
pedagogic criteria for effective resource-based, collaborative learning. 

Academics have used e-mail to communicate with each other since the early 1980's. However 
it is only recently with the growth of the World Wide Web (WWW) and the explosion of the 
internet into popular culture, that many lecturers and academic departments have started to 
exploit the potential of these technologies and the sophisticated network infrastructure 
provided by super-JANET to enhance their teaching. 

Increasing numbers of teaching staff are beginning to put their lecture notes and reading lists 
on the web for students to browse and are communicating with students via e-mail. Course 
outlines are commonly published on departmental web-sites. Some university libraries have 
web-interfaces for searching, checking availability and reserving books. Conferencing 
software is used in some institutions to create on-line discussion groups amongst students. 
These simple innovations are only the beginning and it is not surprising that there has been a 
recent wave of interest amongst software developers and IT research groups within 
universities in exploring ways to further leverage technology in an educational context. 

The systems that have been developed as a consequence of this interest may be viewed as 
specialised GroupWare for education. They are designed to integrate and build on established 
network technologies that have recently begun to be used separately as teaching and learning 
tools. These include conferencing software, e-mail, on-line resources, search engines and 
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multi-media databases, video-conferencing, shared whiteboards and interactive simulations. 

Depending on the pedagogical orientation and intended educational market of the developers, 
these systems are collectively and variously known as on-line learning environments, learning 
management systems, collaborative learning software or virtual learning environments (VLEs). 
For simplicity we use only the term VLEs as a collective term for all the systems described in 
this report. 

1.2 What is a Virtual Learning Environment? 

In this section we sketch out the elements that make up a prototypical VLE. The system that 
we describe below is fictional, but is based on a distillation of the common features and basic 
technology of many of the systems we have looked at. 

Almost all VLE systems currently available are based on a client-server architecture. In 
general the client is simply a web browser that is used to access html pages on the server. 

The server software either sits behind an existing web server or includes its own web server in 
the package. At a minimum the server will be capable of creating and serving up dynamic html 
pages, will allow messages to be posted up to conferences or a web notice-board and will 
maintain a database of information relating to users, groups, learning materials and course 
structure. 

The schematic below indicates the functionality provided by a prototypical system: 



Notice-board 



Course Outline 



E-mail Tutor 
and Students 



Conferences 



Class List & 
Student Homepages 



Assignments 
/ Quizzes 



Assessments / 
Grade-book 



Metadata 



Synchronous 

Collaboration 

Tools 



Multimedia Resources 
Repository 




Calendar 



Search Tools 



Bookmarking 



Navigation 

Modd 



Figure 1.1: A schematic of a Prototypical VLE 

In general, users of a VLE are divided into two main classes: Students and Tutors. Whilst 
tutors have a similar view of the system to students they will usually have additional tools and 
privileges that allow them to add materials, create conferences and track students progress. 
In some cases students have an area for conversation that is private from the tutors view. 

We now briefly discuss each of the features outlined in figure 1.1: 

Noticeboard 

A noticeboard or announcements area is a useful feature that may well appear as soon as a 
student logs in to the system. 

Course Outline 
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The course outline or schedule provides an overview of the course structure and may include 
dates for assignments, assessments, lectures, video conferences etc. Typically the system 
will provide a structured means for the tutor to create the course outline. The course outline 
will provide hyperlinks to the course pages themselves. In a web-based system these will be 
simply html-pages containing the material relevant to that part of the course. 

E-mail 

Most systems include a built in e-mailer that can be used to email either the tutor or individual 
students on the course. 

Conferencing Tools 

Asynchronous conferencing or discussion groups form the heart of many VLEs as they 
provide the means for students to engage in collaborative exchange about topics on the 
course. 

Class List & Student Homepages 

Another key feature of a learning environment is getting to know the other students on a 
course or for tutors to get some idea of students backgrounds, interests and aspirations. Many 
systems incorporate a list of students enrolled on a course - perhaps with e-mail addresses. 
Even more useful is the inclusion of a homepage for each student. Some systems provide an 
easy-to-use front-end for homepage editing. 

Metadata 

Metadata is simply information about an object. It is important for categorising and searching 
objects according to their intended use in a particular context. A sophisticated metadata set 
can encompass a wide variety of information about an object. For example, for a Java applet 
that is included as a resource in a course unit, the metadata would provide information about 
who created it, when it was created, who the target audience is etc. Most current VLEs 
incorporate some simple scheme for providing metadata about resources, course units and 
people, however several systems that are currently under development are paying more 
attention to the metadata issue. To be truly effective any metadata scheme should be 
standardised in some way so that it can be used across different systems rather than being 
local to a particular system. For this reason standards efforts such as the IMS project initiated 
by EDUCAUSE in the US and the Ariadne project in Europe are of particular relevance. 

Assignments 

A VLE should allow tutors to create assignments for students to complete as they work 
through course material. It should provide a means for students to return completed 
assignments to the tutor for grading and feedback. 

Assessments 

Some VLEs provide automated on-line quizzes for performing assessments, self-testing 
versions are also a feature of some systems. 

Synchronous Collaboration Tools 

Synchronous collaboration tools such as Chat, Shared Whiteboards, Group browsing and 
video-conferencing are a feature of some but, by no means, the majority of VLEs. The relative 
importance of such tools in a system depends largely on the intended use of the system 
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Multimedia resources 

One major advantage of VLEs is the ease with which multi-media resources can be accessed 
and stored within the learning environment as an integral part of the course package. As more 
and more sophisticated educational materials such as interactive simulations get published 
on-line, the importance of multimedia facilities will increase dramatically. 

File Upload Area 

For truly interactive functionality students should not just be recipients of content uploaded 
onto the system by a tutor, but should be able to upload their own materials for other 
participants to look at. Some VLEs include a facility for students to build their own materials 
and objects they have found into the learning environment. 

Calendar 

A Calendar tool is a useful feature built into some VLEs. 

Search Tools 

When a course structure becomes very large or there are several participants navigating 
around the environment by browsing and hyperlinks alone can become quite cumbersome. 
Consequently some systems incorporate search tools in order to jump straight to subjects of 
interest or a particular person. 

See also the section on the Navigation model. Search tools are particularly useful when a 
large resource base of materials is built up within the VLE. 

Bookmarking 

Like search tools, a bookmarking facility can significantly decrease the amount of time spent 
navigating to frequently used places or items within the environment. Some systems include a 
more sophisticated version of bookmarking that allows participants to build up their own 
individual resource base. 

Navigation Model 

Although navigation is not strictly a feature or tool within a VLE, it is intrinsically part of the 
experience of using a VLE. The navigation facility allows a user to move around the 
environment and the navigation model or metaphor in conjunction with the look-and-feel of the 
system is extremely important as it defines in many ways how the system is used. In addition 
to using hyperlinks and page to page browsing which are common to the experience of using 
a normal web browser, different VLEs will present the tools available and course structure in 
different ways. Two popular models are to use a homepage for the course which is presented 
on log-in with hyperlinks that act as jump-stations to the various tools that are available or, 
alternatively, to use a hierarchical tree structure. The two are often used in conjunction with 
the tree structure providing a course outliner with links to the course content packed into the 
branches of the tree. 

It should be emphasised that although we have provided an overview of what to expect from a 
VLE, we have only described the most prevalent current model of a VLE architecture above, 
that is the Web-client and Server approach. This may well change in the near future as 
technologies are becoming available that allow different architectural model to be used. For 
example at least one of the systems we describe in appendix 1 of this report (Learning 
Landscapes) differs markedly from this model and uses a completely distributed approach in 
which each client is a stand-alone Java application that incorporates a Web-browser within it. 
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It should also be noted that we have only included a subset of the tools and features that may 
be available in any given VLE system. In appendix 1 we provide a more detailed description of 
a number of VLEs that are currently available (Tours of each of these systems are also 
provided in the CD accompanying this report). In appendix 1, we also provide a breakdown of 
the features offered by each of the systems. 

1.3 Currently available VLEs 

The VLE systems that are currently available originate from either of two main historical 
sources and is important to be aware of the difference. The first source, which accounts for 
the majority of commercial systems on the market is the on-line distance-learning sector in the 
US. Popular VLEs such as TopCIass, LearningSpace and Web Course in a Box are examples 
of systems that have developed from this background. Distance-learning, however, is only 
part of the reason for current interest in VLEs as institutions seek ways to use technology to 
make teaching more effective on-campus as well. It is not clear that systems designed with 
distance learning in mind are best suited to this purpose. There are a number of systems 
currently under-development within UK universities that are aimed at both on-campus and 
distance learners, (CoMentor, Learning Landscapes, CoSE). 

There is undoubtedly a broad selection of different systems that have been developed or are 
currently in the process of being developed that would qualify for inclusion in a list of current 
VLE systems. However, an exhaustive review of systems is neither practicable nor would 
remain exhaustive for very long. Thus we have selected a manageable number of what may 
be regarded as the 'field leaders' that form a representative sample of the various tools that 
exist from both commercial and Higher Education sources to use as example systems in this 
report. The initial selection criteria were based on reputation, the amount of functionality 
provided by the tool, the technology infrastructure required to use the tool, its applicability for 
use in Higher Education and in particular its relevance for UK universities. 

Table 1.1 below lists all the systems that are used as examples in this report, with URLs to 
their WWW homepages. Appendix 1 describes these systems in more detail. 
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Product 


Organisation 


URL 


Learning space 


Lotus Education of Lotus 
Institute 


http://www.lotus.com/ 


WebCT 


WebCT, Univ. British 
Columbia 


http://www.webct.com/ 


TopCIass 


WBT Systems 


http://www.wbtsystems.com/ 


Virtual -U 


Virtual Learning 
Environments Inc. 


http://www.vlei.com/ 


Web Course in a Box 


MadDuck Technologies 


http://www.madduck.com/ 


Asymetrix Librarian 


Asymetrix 


http://www.asymetrix.com/ 


FirstClass Classrooms 


SoftArc 


http://www.softarc.com/ 


Courseinfo 


Blackboard Inc 


http://www.softarc.com/ 


ARIADNE 


EPF Lausanne (EC DG XIII) 


http://ariadne.unil.ch/tools/ 


CoMentor 


Huddersfield University 


http://comentor.hud.ac.uk 


CoSE 


Staffordshire University 


http://www.staffs.ac.uk/cose 


Learning Landscapes 


TOOMOL Project, UW - 
Bangor 


http://toomol.bangor.ac.uk 



Table 1.1; Virtual Learning Environments 



1.4 Who uses VLEs? 

In order to estimate the current usage of VLEs in UK Higher Education We conducted a 
survey of 100 UK Higher Education institutions using the instrument copied in Appendix 3 to 
determine the current usage of online systems for learning and teaching. The purpose of this 
survey was to discover which (if any) of the integrated VLE packages are currently being used 
and if so to obtain some feedback on their ease of use for performing various tasks. We were 
also interested to find out how many respondents reported using any on-line system for 
teaching and learning (e.g. putting lecture notes on the world wide web). 

In the survey instrument we first asked respondents to describe any systems that are used for 
teaching and learning. In particular we asked for details of the use of virtual learning 
environments, conferencing systems, the world wide web and synchronous communications 
systems. The second part of the instrument queries respondents on the effectiveness of 
systems they use in a number of tasks. These were grouped according to the tasks that a 
VLE may be expected to assist with. The survey was then sent to the information services 
department of each institution. 

The results of the survey were as follows: 

Out of the 100 survey forms sent out eleven were returned by the cut off date. 
Out of the eleven responses: 
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o All eleven reported using the WWW in some form for teaching and learning, 
o Nine respondents reported using a virtual learning 
o Five respondents reported using the First Class conferencing system, 
o Six reported using some form of synchronous collaboration system, 
o Three respondents reported using virtual labs or collaborative simulations. 

The number of survey forms returned was unfortunately too low to warrant detailed analysis of 
the results and any conclusions drawn from these results are necessarily speculative. 
Possible reasons for the low rate of return other than simple lack of willingness or time to 
participate are: 

o The surveys were sent to information service departments. However it is possible that 
they would not know about systems being implemented by academic departments or 
individual lecturers. Informal evidence from a number of institutions suggests that few are 
currently attempting to implement a co-ordinated solution for the whole institution, rather 
many different solutions have been put into operation by enterprising departments and 
enthusiastic individual lecturers. This situation raises interesting questions about the use 
of VLEs to support existing coursework in institutions. It may not be an appropriate model 
for institutions to purchase a single heavyweight system to attempt to cater for the needs 
of all departments as different departments and lecturers have different requirements. 
This issue is addressed in the next section. 

o The use of virtual learning environments and other on-line approaches to learning is still 
embryonic in many institutions. This may be expected to change rapidly if the high level 
of interest and awareness amongst educators about VLEs and alternative teaching / 
learning methods and the high rate of development of new and improved systems 
persists. 

1.5 Why are VLEs relevant to Higher Education in the UK? 

Despite the hype that inevitably surrounds anything on-line, 'virtual' or web-based at present, 
there are some clear and immediate benefits of these systems to students and teachers alike 
that concord with the recommendations of the Dearing Report (1997) regarding C&IT use. 

o Flexibility of time and place, 
o Coping with increased student numbers, 
o Sharing and re-use of resources, 
o Collaborative work, 
o Student-centred learning, 
o Reducing the administration burden, 
o Staff Development (Milligan, 1998) 

Many advocates of VLEs within Higher Education are excited about their use because of their 
potential to allow a resource-based and student-centred approach to learning to be 
incorporated into their teaching, (e.g. Collis, 1996). One of the major blocks to adoption of this 
style of teaching and learning in universities is the extra time burden it places on tutors over 
the more traditional content-centred approach. Perhaps the key contribution that VLEs can 
offer is to allow a resource-based approach whilst also alleviating the tutor from an extra 
administrative burden. 

There are various ways that VLEs might achieve this but as yet there is currently no 
established evaluation methodology from a pedagogical perspective with which to assess 
different systems. Most discussions and evaluative reviews of VLEs to date have tended to 
concentrate on the features, technical details and pricing of different systems. Whilst this 
information is invaluable in determining the choice of system, we argue in this report that an 
overriding concern is matching the pedagogical orientation of the system with its intended use 
in teaching and learning. 
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Mason (1998) suggests that there are three basic models of existing on-line courses 

o Content + Support Model. Here a relatively static body of content (e.g. a web package) 
provides the core of the course and is supplemented by tutorial support. The level of 
on-line interaction is low (typically no more than 20% of the students time). This is the 
model that is most akin to traditional teaching and is the most prevalent model currently in 
use. 

o Wrap-around Model. Here the course materials are wrapped by activities, on-line 
discussions etc. Mason refers to this as a 50/50 model as on-line interactions and 
discussions occupy roughly half the students' time, 
o Integrated Model. This is a resource based model where the course is defined by 
collaborative activities, discussions and joint assignments. The course contents are 
dynamic and are determined largely by individual needs and group activities. Resources 
are contributed by participants or tutors as the course develops. 

This provides a useful framework within which to consider the role of VLEs, and CAL systems 
in general, in UK Higher Education. Historically, and indeed until very recently, online courses 
and other CAL modules have consisted of either the content + support model or the 
wrap-around model. It is only with the development of fully integrated VLEs that implementing 
the integrated model on-line is a real possibility. 

The first two models either replicate the structure of a traditional taught course or can be 
inserted into a course as a component without requiring any major change in approach from 
either staff, students or institutional infra-structural support. This represents a major difference 
with the integrated model. 

Use of an integrated VLE implies that both staff and students are prepared to adopt an 
educational approach that is inherently more pro-active from the students perspective and that 
is in turn less pre-structured and more responsive to students requirements from the teachers 
perspective. Thus, first and foremost an evaluation framework needs to relate to the 
pedagogical process that the VLE is intended to support. 

Aside from these educational factors there are administrative and management concerns 
associated with adoption of an integrated VLE, which cannot and should not be ignored. In 
order to maintain focus on the primary aims of this report, discussion of these factors is 
deferred until the end of the report. 

1.6 Developing an evaluation strategy 

As we have already mentioned, many comparative evaluations of software are carried out in a 
rather reductionist way, based on numbers of features, as opposed to looking at the function 
and usability of the overall system in the context of the human or organisational systems 
within which it is to be embedded. In the table in appendix 2 we provide a summary of the 
feature sets of all the systems outlined in this report. The asterisks that appear in the table 
denote a function that is implicitly available within the system even if it is not a feature. The 
information contained in this table is drawn from the manufacturer's or developer's own 
documentation, existing reviews and reports and personal experience of the systems. There 
currently exists a small number of comparative reviews of existing VLEs based on feature 
comparisons. Probably the most comprehensive of these is one conducted by Bruce Landon 
at the University of British Columbia. This is an on-line extensible review to which contributed 
data may be added. URLs for this and other online reviews are provided in the bibliography. 

What becomes immediately apparent from an examination of the feature table in apperidix 2 
is that it does not distinguish a great deal between the different systems. The question is not 
as simple as deciding which system offers the most functionality for your money. Most of the 
systems can accomplish most of the tasks identified in the 'tools' column. However, this type 
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of table says nothing about how easy or difficult it is to carry out a particular task. Moreover 
many of the systems are capable of implicitly supporting functionality by virtue of the web 
environment or the addition of third-party plug-ins. A good example is FirstClass. Although it is 
primarily a conferencing system, because of the web environment it is implicitly capable of 
providing an entire structured on-line course by authoring the appropriate web-pages and 
providing links to the discussions from them. 

Most of the systems reviewed would be capable of supporting a content-driven model of 
online teaching and learning mentioned earlier (although in this case we might ask how much 
added-value they provide over simply using the web and e-mail?). What is less clear from the 
feature summary is how well different systems are suited to sub-serving the wrap-around or 
integrated models in the framework put forward by Mason (1998). 

The answer, to our minds at least, lies not solely in the features of a system, but in how they 
are integrated to facilitate learning and administration and what metaphors are constructed to 
guide the way the system is used. In short we suggest that a more holistic approach to the 
evaluation of these packages is needed. Our aim is to ascertain the educational principles 
around which the system was designed and how well it fulfills our requirements as educators 
or learners. 

The main thrust of this report then, is to propose an evaluation strategy from a pedagogical 
perspective. We have identified two key considerations for VLB evaluation strategies: 

o VLEs should provide opportunities to improve the quality and variety of teaching and 
learning that are not being achieved using current methods, 
o VLEs should reduce the administrative burden on teachers, thus allowing them to 
manage their workload more efficiently and to be able to give more time to individual 
students educational needs. 

In order to address these issues, we consider two possible models upon which an evaluation 
strategy may be based. The first, and most obvious choice, is the Conversation Framework 
proposed by Laurillard, (1993) and one author Crawley (1999) has very recently published a 
paper in which an evaluation methodology for collaborative learning tools which is based on 
this model is proposed. The other is a model drawn from cybernetic approaches to modelling 
organisations known as the Viable Systems Model (Beer, 1981). The VSM provides a 
framework for us to think about the educational context into which a VLB is being placed and 
how the system helps a tutor manage the complexities of teaching a resource-based course 
with a large number of students. 

2. Evaluation of VLEs using the Conversational 
Framework 

In this section we explore the use of the Conversational Framework developed by Laurillard 
(1993) as an evaluation methodology for virtual learning environments. 

We have already referred to the notion of a learning conversation (Laurillard, 1993; Dearing 
Report, 1997) as an alternative to a more traditional delivery approach to teaching. Whether 
VLEs are being used for distance learning or to enhance learning within institutions, it is our 
view that their most interesting role is as a medium for supporting constructivist and 
conversational approaches to learning. Thus Laurillard's conversational model offers itself as 
an obvious and interesting candidate for providing the basis of a pedagogical evaluation 
framework for VLEs. 

The roots of the model lie in the Conversation Theory developed by Gordon Pask (1976), 
although Laurillard traces the need for dialogue in the learning process back to the Socratic 
method of philosophical enquiry. The centrality of dialogue in the model comes from the need 
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for the teacher to unearth the student's mental constructs about a topic before negotiating the 
path to the target conception that is the goal of learning from the teacher's perspective. 

The teaching strategy advocated in the model is based on the form of the interaction between 
teacher and student and not solely on the actions required of the student. The model 
advocates that action on the part of the student is constructed around the dialogue and should 
be supplemented by constructive and meaningful feedback from the teacher. Additionally 
there should be opportunities provided for student reflection. 

2.1 The Conversational Framework 

There are a number of key characteristics of the conversational model as applied to academic 
learning. These are drawn from Laurillard, (1993, pp. 94-95). 

Discursive 

o Teacher's and student's conceptions should each be accessible to the other 
o Teacher and students must agree learning goals for the topic and task goals 
o The teacher must provide an environment within which students can act on, generate and 
receive feedback on descriptions appropriate to the topic goal. 

Adaptive 

o The teacher has the responsibility to use the relationship between their own and the 
student's conception to determine the focus of the continuing dialogue. 

Interactive 

o The students must act to achieve the task goal 

o The teacher must provide meaningful intrinsic feedback on the actions that relate to the 
nature of the task goal. 

Reflective 

o The teacher must support the process in which students link the feedback on their actions 
to the topic goal for every level of description within the topic structure. 

The conversational model as shown in the diagram of figure 2.1 depicts the workflow between 
tutor and student during learning. Certain activities (centre in blue boxes) are interactive and 
take place through some medium. Other activities (right and left in yellow boxes) are internal 
to either the student or the teacher. If we suppose that the medium involved is a VLB, then 
this model provides a clear set of requirements for evaluating the system's suitability for 
supporting the processes that form the basis of interactive learning. 
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MEDIUM 



STTTDF.NT 




Figure 2. 1 The Conversational Framework (adapted from Laurillard, 1993) 

The primary workflow actions that take place through the interactive medium are as follows: 

1 . Teacher presents / redescribes conception 

2. Student presents / redescribes conception 

3. Teacher sets up micro-world activities 

4. Student interacts with micro-world activities 

5. The system provides feedback on the action 

6. Student modifies actions in light of feedback 

2.2 The conversational model and VLEs 

This relatively simple framework offers much potential as a methodology for evaluating virtual 
learning environments. A similar proposal has been recently put fonward by Crawley, 1999. In 
order to make this scheme work, all interactions must be assumed to take place via the 
medium of the VLE. It is then possible to make judgements about how well the VLE handles 
each of the individual interactions labelled in the model. The sort of questions one might ask 
are: How does the learning environment allow a teacher to present a conception? What does 
it mean for a teacher to set up a micro-world within a VLE? How can the student interact with 
the micro-world? 

The model raises further questions about the mechanisms that support conversations. How 
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easy is it to track conversations relating to a particular issue? Can conversations be enhanced 
by presentation of additional resources? Yet other questions relate to the overall flexibility of 
the system. How adaptable are micro-world structures once they are in use? How easy is to 
tailor them to individual students needs? 

We might also want to ask whether the learning environment provides any extra tools to 
support reflection by the student on the relationship between the conversation at the level of 
descriptions and the activities they have completed or whether it provides structuring to assist 
the student in modifying their actions. 

The interactions illustrated in the model do not necessarily occur in the linear order given 
above and the model involves recurrence at several levels. In particular the conversation may 
shift freely between the conceptual level and the level of actions. Any software environment 
supporting learning should allow this to occur naturally. 

2.3 Evaluation criteria for VLEs 

In order to evaluate a virtual learning environment using the conversational framework we 
need to establish what tools are provided within the software to allow dialogue and action to 
mutually influence each other to allow modification of both conceptions and actions on the 
part of the student as described above. 

Additionally we need to be able to say what level of structuring is provided for each 
interaction. For example if e-mail is used as the tool for communication between tutor and 
student in a given VLE we need to know how the use of e-mail is embedded within the context 
of the dialogue about a specific topic or set of learning goals. Thus we might consider whether 
it is: 

a. simply a link to the users own e-mail client and existing mailbox 

b. a built-in e-mail client that allows messages to be stored and viewed from within the 
software 

c. a fully integrated client that automatically builds a conversation thread that is easily 
located from material relating to the specific topic of discourse. 

Clearly b) and c) would be more sophisticated tools from the pedagogical perspective of the 
conversational framework than a). 

Another issue that quickly becomes apparent is that the notion of a 'micro-world' takes on a 
different meaning in the case of VLEs than more traditional forms of courseware. In essence, 
the VLE provides the tools for a teacher to build a micro-world by allowing the teacher to 
construct learning activities enriched by multimedia resources and simulation programmes. 

The crucial point from the perspective of the conversational framework is that the teacher 
should be able to construct the learning activity following a conversation with the student at 
the level of conceptions and the identification of a learning goal for the topic in question. Thus 
for any given learning environment we should consider how well the tools provide for both 
structuring conversations and actions and also how well they allow for integrating dialogue 
with actions. 

Discursive Tools 

All of the VLEs described in appendix A contain tools for conducting conversations. Naturally 
these rely to a great extent on e-mail and asynchronous discussion groups. It is important to 
consider how well the VLE leverages e-mail technology to support the conversation as an 
integral part of learning. For example, is the conversation accessible directly from the learning 
topic within the course structure or does the user have to move out of the course work in order 
to continue the conversation? Does the e-mail or conferencing tool allow attachments to be 
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included with messages? And if so, can the attachment be extracted and embedded into the 
users course work structure? Does the tool allow learning goals to be specified and recorded 
on the basis of the conversation? Ideally the agreed learning goal should be in a prominent 
location with respect to the topic of learning. 

Adaptability 

How easy is it to adapt the activities associated with a learning topic according to the needs of 
an individual student or student sub-group as revealed by the conversation? This raises the 
question of whether students in the same group can be easily differentiated within the VLE 
once a course or learning activity is in progress. 

Interactivity 

A basic pre-requisite for a VLE is that it should be interactive. It is not enough for material to 
be presented to a student and then be tested on it. A VLE should allow the students to 
restructure the presented material, add resources of their own, annotate material, launch and 
run simulations etc. In other words the student should not merely be a passive observer of the 
'micro-world' constructed by the teacher, but should be pro-active in shaping the 'world'. 

Reflection 

How does the VLE allow the teacher to help the student link detailed feedback on their actions 
to the topic goal? A concept-mapping tool might be a helpful feature in this respect. 
Alternatively, contextualised discourse for every level of the topic structure should be possible. 

2.4 Constructing an Evaluation Framework 

A suitable evaluation framework for VLEs using the conversational model could be 
constructed in a variety of ways. The first way that we suggest involves constructing a table 
that describes the tools that are available for each of the stages of interaction described in the 
model (see Table 3.1). In addition, it is helpful to also describe what support is provided for 
structuring the conversations and activities. This is an important element of the evaluation 
framework because it helps to describe how well integrated the tools are. Most of the tools 
that are incorporated into VLEs can be found separately in other packages, thus in some 
ways the very attraction of a VLE is the integration it provides. 

Table 2.1 below illustrates the evaluation framework for VLEs using the interactions in the 
conversational model as criteria against which to identify the tools and level of structuring 
provided by the VLE. For each interaction we have provided examples of what we might look 
for in an integrated VLE. 





Tools 


structuring 


1. Teacher 

Presents 

Conception 


What tools does the teacher 
have to hand; 

Text, video, audio, images? 


Can a teacher easily put 
together different 
multimedia formats for 
presentation of a 
conception? Can these be 
readily altered for 
re-presentation in a 
different way 






Clearly the dialogue 
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2. Student 

Presents 

Conception 


Can the student interact with 
the teacher through the 
system? Does the student 
have multimedia authoring 
capabilities? Even if 
text-only, how does the 
student communicate with 
the teacher? 


between student and 
teacher is at the centre of 
the conversational model 
and how this is visually 
structured for both tutor and | 
student is very important. 
Conversations should be at j 
the centre of activity in the 
VLE rather than pushed to 
one side. 


3. Teacher sets up 
micro worid 


Multimedia authoring tools 
for creating course 
materials, embedded or 
linkable simulation 
programs, testing software 
such as quiz creation 
programs etc. 


In a VLE the notion of 
micro-world can be applied | 
at many different levels. 

The important point from 
the perspective of the 
conversational model is 
that it should be versatile 
enough to be adapted for 
an individual student on the | 
basis of the ongoing 
conversational dialogue 
with that student. 


4. Student 
interacts with 
micro worid 


See 3 above 


Again we can see this 
notion of micro-world at 
various levels. We are 
looking for more from the 
student side than simply 
being able to view content. 


5. Tutor provides 
feedback to the 
student 


Can the tutor use the 
communications tools to 
provide feedback to the 
student in the context of the 
students' activities? 


It might seem obvious that 
this would be true but the 
important point is that the 
feedback can be easily 
related to the action - i.e. 
any discussion thread 
should be linked to or 
embedded in the domain of 
actions. 


6. Student 
modifies actions 


Can the student return to the 
activities and modify their 
actions based on feedback 
received from the tutor? 





Table 2.1: An evaluation framework for VLEs using the conversational model 

2.5 Testing the evaluation framework 

We implemented this methodology for several of the VLE systems described in appendix 1 . In 
most cases we only had access to demonstration versions of the software systems, on which 
to perform evaluation tests. Since many potential evaluators will also only have access to the 
demonstrations that are available, we felt this was a realistic test scenario. Evaluations for 
two of the example systems are illustrated below. 
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VLE1: WebCT 





Toois 


structuring 


1 1. Teacher 
Presents 
Conception 


Primary presentation is 
through course material 
space (html pages). Other 
presentation of concepts is 
possible via e-mail, Bulletin 
Board and Whiteboard 


Learning Goals can be set 
for each page of content. 


2. Student 

Presents 

Conception 


Student Presentation Area 
permits uploading of student 
materials. Othen/vise e-mail, 
BBS and Whiteboard 




3. Teacher sets up 
micro worid 


The micro world essentially 
is the course structure and 
materials. These are mostly 
prepared in advance. More 
fine grained materials can 
feasibly be inserted at this 
stage. 

Any Web-interactive content 
can be incorporated 


Dynamic tree 
representation allows 
hierarchical structuring of 
course 'path'. Each tree 
branch is a web page. 
Calendar Tool allows time 
structuring. 


4. Student 
interacts with 
micro worid 


• Browsing content 

• Auto - Quizzes 

• Annotates content 

• Interacts with 
simulations and other 
live content 


Study guide generation 


5. Tutor provides 
feedback to the 
student 


e-mail 

BBS (full conferencing tool) 
Whiteboard 




6. Student 
modifies actions 


Student can freely modify 
actions 





VLE2:Virtual-U 
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Toois 


Structuring 


1. Teacher Presents 
Conception 


Course Space within the 
campus provides for initial 
presentation of ideas by the 
teacher. Subsequent 
presentation would occur in 
the Conferences Space or 
by private e-mail. 


Assignments can contain 
learning goals 


2. Student Presents 
Conception 


E-mail to the teacher or 
posting to a conference. 
Submission of assignments 




3. Teacher sets up 
micro worid 


The micro-world afforded by 
Virtual-U consists of many 
features: The course itself, 
the Gallery, the Library as 
well as assignments and 
activities associated with 
the topic. 


There are online help files 
to assist instructors with 
setting up different kinds of 
micro-worlds, assignments 
and projects with Virtual-U 


4. Student interacts 
with micro worid 


The Campus metaphor 
provides a rich environment 
for student interaction with 
the system. Navigation 
between spaces is easy 
and flexible to use. 


There is a Glossary and 
extensive online help files to 
assist student working 


5. Tutor provides 
feedback to the 
student 


The tutor has access to a 
wide variety of information 
about student activities and 
progress. One-to-one 
feedback online would be 
given via e-mail or a 
conference of only two 
participants could be 
created. 




6. Student modifies 
actions 


Student can freely modify 
actions 





One problem that we discovered when using the framework as described above for the tests 
was that several of the stages of interaction identified for the conversational framework 
require the same information regarding tools. It would be desirable to compress this 
information in some way. Also there is a semantic peculiarity that is thrown up by stages three 
and four - setting up and interacting with a micro-world. In the case of a VLE these stages 
really imply simply using the VLE. The important point is that the activities and structures 
set-up in the VLE by the teacher should be adaptable on the basis of prior discussion. 
This alone is a powerful discriminator of VLEs that are designed for delivery as opposed to 
those that are intended to be truly interactive 

An alternative way of constructing the evaluation to take these factors into account would be 
to build the table using the principles rather than the stages of the conversation model: 
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discursive, adaptable, interactive, reflective. We have implemented this strategy below for two 
further systems. 

VLE1: TopCIass 





Tools 


Structuring 


Discursive 


E-mail and discussion lists 
comprise the communications 
tools 


Both are accessible from the 
course homepage. 
Discussions can be 
integrated with course topics 
in TopCIass server 3. 
Attachments can be included 
in e-mails to aid presentation 
of a conception. 


Adaptable 


Units of Learning material 
(ULMs) allow hierarchical 
structuring of course content. 


Very flexible structuring 
capability allowing individual 
assignment of learning 
content. 

Date and time restrictions 
can be added to control 
access to course materials 
located in ULMs. This 
includes Tests 


Interactive 


The interactivity of TopCIass is 
largely determined by the 
content that is placed in it and 
the way both course content and 
discussions are structured by the 
teacher 


There is no provision for 
students to add their own 
materials and resources into 
the course structure provided 
by the tutor. 


Reflective 


Students can view their own test 
scores. Feedback can be 
provided through e-mail and the 
discussion lists. There is no 
off-line working facility which 
would prove a restriction on 
students ability to access their 
work in the absence of a network 
connection 





VLE2; COSE 
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Toois 


Structuring 


Discursive 


People in COSE are organised 
into groups which can be 
hierarchical. E-mail can be sent 
to the group from within the 
group manager tab. There is 
also a noticeboard for notices 
from group managers. There is 
no discussion group tool. 


COSE is organised around the 
sharing of work and has 
sophisticated tools for the 
creation and sharing of 
content, discursive tools 
however are limited. To 
compensate for this there is a 
great amount of flexibility in 
the creation and management 
of groups which means that 
conversations can easily be 
contextualised to the specific 
topic of interest. Unfortunately 
e-mails cannot be read and 
stored within COSE, so a 
conversational history cannot 
be built within the system. 


Adaptabie 


Learning opportunities can be 
created easily and resources 
can swiftly be added by using 
the Gatherer, the Basket or the 
publishing tool. A learning 
opportunity can then be 
assigned to any group or 
subgroup 


The tools for re-use of content 
are excellent and once the 
tool has been in use for a 
short while individualised 
learning opportunities could 
be very quickly created and 
assigned. These tools are a 
major feature of the COSE 
design 


interactive 


COSE is a highly interactive 
environment. The emphasis is 
on sharing work-in-progress 
with other members of a group. 


The facility for learners to 
create and manage their own 
groups and sub-groups means 
that the burden of work shifts 
from the tutor to the student 
as they become pro-active in 
shaping conversations and 
activities. 


Refiective 


By virtue of the organisation of 
the system around people 
rather than content, COSE 
allows feedback to be easily 
targeted at the right level and to 
be supplemented if necessary 
by additional resources drawn 
from the Basket. There is no 
off-line option with COSE. 





2.6 Summary 

The examples provided above show how the conversational framework originally proposed 
by Laurillard might be adapted to serve as a way of evaluating VLEs. We have considered 
two methods of breaking down the elements of the model in order to assess the 
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correspondence of a VLE to the requirements of the model, although undoubtedly others are 
possible. 

In the first pair of examples we used the different stages of interaction as criteria against 
which to compare the system tools. This is useful because comparison of tools for stages 1 
and 2 (teacher presents conception and student presents conception reveals the 
communication channels open to BOTH teacher and student and whether there are any 
special tools to assist with this process, such as a shared whiteboard for example. 

The third and fourth stages deal with the level of actions in the conversational model. As we 
have stated, the 'micro-world' essentially is the VLE and it determines how much flexibility 
and scope there is for the teacher to set up and assign projects and activities within it. There 
are a wide range of approaches and some systems will be more suited to setting up the 
course structure in advance whilst others such as COSE are tailored towards setting up 
activities on the fly. The metaphor of the VLE has significant impact on how stimulating and 
easy to navigate the system is. WebCT, TopCIass and several others adopt a simple strategy 
of linking tools to a course homepage. Virtual-U elaborates on this notion by providing a 
campus metaphor. Co-Mentor also builds on the notion of places to increase the contextual 
cues within the system. COSE has a detailed interface with a vista of the various projects and 
tabs denoting the various tasks such as gathering things and putting things in your basket. 
These metaphors not only affect the experience of using the system, but directly influence 
how teaching and learning is performed. 

In the fourth stage, 'student interacts with micro-world' we considered how much scope there 
was for the student to shape their own view of the projects, to interact with the content and to 
add their own resources. Again systems vary quite considerably in this respect and this may 
be revealed here. 

Stages 5 and 6 are less useful in the context of VLEs because there will, in essence, the 
same tools for a tutor to provide feedback as there were to engage in the initial conversation. 
Also due to the nature of a VLE in contrast to other types of educational software, the extent 
to which students are able to modify their actions is determined more by constraints laid 
down by the tutor than by the software. 

Because we have used the conversational model somewhat differently in the context of 
VLEs, we formulated a second, simplified version of the evaluation table using the principles 
of the model as criteria. Although this version is more abstract in terms of the specifics of the 
model, it proved easier to use and draws out the crucial distinctions between systems at least 
as well as the previous version. We illustrated this approach using TopCIass and COSE as 
sample systems. TopCIass emerged as an all-round system that is more suited to a 
traditional style of on-line course. COSE, by contrast, is centred around collaboration in the 
sense of sharing content amongst groups of learners and the publication and re-use of 
materials. The conversational element in COSE has been pared down to a minimum. 

One of the problems with the evaluation of software based on feature comparisons is that it is 
easy to lose sight of the wood for the trees. In this chapter we have presented the 
conversational model as a way of attempting to capture key aspects of a system from within 
a framework of educational 'good practice'. 

However one of the main limitations of the conversational model is that it fails to reflect 
functionality associated with managing groups of learners. In the section we consider an 
alternative model with that aim at its core. 

3. A Cybernetic Model for evaluating Virtual Learning 
Environments 
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3.1 Introduction 

A key issue, perhaps THE key issue, for VLEs is concerned with how to allow for the 
management of groups of learners, for example when the VLE is being used to support 
traditional courses (ranging from 10-100+ students?). Formal teaching has a well evolved set 
of tools and approaches for dealing with groups of students en masse: these include lectures, 
seminars, tutorials and laboratory sessions. People are involved in the system in different 
roles - lecturers, postgraduate demonstrators, and personal tutors amongst others. All of 
these mechanisms and people are integrated and organised by a timetable, which constrains 
how and when they can take place; and how they fit into a larger scenario involving many 
courses and demands on space and time. 

If VLEs are to be successful in enhancing educational quality, or if they are to achieve greater 
efficiency than traditional methods, they need to be able to address the types of issues that 
have previously been dealt with by the methods, techniques and tools outlined above. To be 
able to understand how they may attempt to do this requires some model that can generalise 
and explain these issues formally. 

We have found Stafford Beer's work in management cybernetics extremely useful in 
suggesting such a formal approach, particularly his Viable Systems Model (VSM). This 
proposes that the structure of an organisation is primarily concerned with managing the 
complexity of interactions between its members, and with other stakeholders, in maintaining 
its identity and purpose(s) over time. The roles of organisational members define which 
communication channels are available to them, and with whom they can interact. In a highly 
hierarchical organisation (e.g. the army), the only valid communications are those with 
immediate superiors or immediate subordinates; in looser organisations (like universities) 
more communication options are available depending on committee structures and their 
membership. 




V (Environment) »> V (Organisation) 



Figure 3. 1: The organisation in its environment 

[NB. In figure 3.1, V stands for variety, the measure of complexity. In formal terms, it is the 
variety of the environment that is much larger than the variety of the organisation. Hence the 
organisation must amplify its variety or attenuate the environment's.] 

The key problem for an organisation in maintaining its viability is how to manage the 
complexity of the situation it finds itself in. It is self evident that the world (or market) that any 
enterprise inhabits is much more complex than the enterprise itself, and yet somehow this 
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has to be dealt with without the organisation being oven/vhelmed. How it does this determines 
the type of organisation it is. 

It could put a lot of effort into understanding the environment, and try to respond to its 
perceived requirements (e.g. market research): or it could put a lot of effort into trying to 
shape the environment into something that suits the enterprises needs (e.g. advertising). 
Most likely it would need to adopt a range of approaches; but success is crucial. The most 
powerful but most devastating approach to managing complexity is to IGNORE it. 

This elaboration also applies within an organisation: management needs to manage the 
complexity of the workforce or operations. The full picture becomes this: 




Figure 3.2: The Simple Viable System Model 

Again there are a range of approaches that have been adopted in organisations. One is to try 
to constrain this potential complexity by limiting the legitimate actions of the workforce 
through strict job definition and demarcation - the Taylorist model. The problem with this is 
that organisational flexibility is lost, and the abilities of the workforce are not fully exploited to 
the benefit of the enterprise. Another is to look for ways in which operations can become 
self-managing while remaining within the overall guidance of management. This requires a 
number of communication channels between management and operations for specific tasks. 
These include: 

■ Resource negotiation 

■ Coordination 

■ Monitoring 

There also need to be communication channels that permit self-organisation and allow 
different operational elements to interact with each other without involving management. 
Self-organisation allows the operational elements to soak up much of their own complexity. 
How these channels are constructed, determines organisational flexibility and effectiveness. 
The schematic diagram in figure 3.3 indicates how self-organisation amongst sub-systems in 
an organisation can reduce the load on communication channels with the management 
without having to introduce a fixed hierarchical architecture. 
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Environment 




Figure 3.3: The Viable System Model: key operational communication channels 

It is not the place here to go into much more depth about the VSM, we shall expand more on 
the relevance of figure 3.3 to education in the following section. However, there is one more 
aspect that will later be shown to be highly relevant to this report; that is the concept of 
recursion. The VSM is particularly powerful because it suggests that organisations can be 
seen as consisting of smaller BUT POTENTIALLY VIABLE organisations, working together to 
achieve mutual benefit. Each of these will also have to handle the same types of 
organisational issues, and so the VSM can be applied at multiple levels to understand and 
design how they function. The answers may be specific to each case - but the questions are 
the same. 

3.2 The VSM and education 

Education, (or at least formal education) depends on organisations. Some have formal and 
independent status (e.g. universities) others are more notional (e.g. the education system). 
Some last for may years, others come and go (e.g. courses). However, each is susceptible to 
interpretation using the VSM. Virtual Learning Environments represent both threats and 
opportunities for universities (and other educational establishments) - threats to traditional 
certainties, opportunities for increased effectiveness and efficiency. We need to understand 
VLEs in organisational terms to properly appreciate the types of impact they can have, and to 
be able to exploit them for our purposes. 

We suggest that VLEs can be examined at 2 levels of recursion: the course level, and the 
institutional level. To date most VLEs have focused at the former, but increasingly they are 
being seen as providing institution wide solutions. However, the main focus of this report has 
to reflect the current state of the art, and so focuses at the level of the course. Before we 
proceed, we have to make an assertion that affects the forthcoming elaboration. We view 
learners as being the workers in the educational enterprise, and not as customers. Of course they 
have aspects of both, more so since they began to pay directly for access to courses at universities. But 
from our point of view, it is learners who have to work to create the change in themselves that we call 
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learning. This process is facilitated, resourced, co-ordinated and monitored by teachers, but is 
undert^en by learners. So applying the VSM at this level we get this picture; 



Knowledge Domain 




Figure 3.4: the VSM applied to a course 

The questions we need to address are how the organisational framework that enables the 
operation and adaptation of the courses does so. 

Resource negotiation : how do learners negotiate their learning 'contracts' with their teacher? 
IS this a one off or a continuous process? What are their mutual rights and responsibilities? 
What is the currency of this negotiation? 

Coordination: can learners collaborate in creating their learning? How? What provision is 
there that can prevent exploitation? 

Monitoring: how does a teacher monitor whether learning is happening, so that, if necessary, 
remedial action can be taken? 

Individualization: how can each student find their own resources and advance their own 
learning independently of others? Can they contribute their discoveries to the group? 

Self-organisation: what space or tools are available to let the learners organise themselves as 
a group, outside of the teacher's purview? 

Adaptation : is it possible for the teacher to adapt the course and its resources in light of 
experiences gained during its operations? 

These questions are equally valid for the organisation of formal teaching and for virtual 
learning: it is the answers that are different. Although there is considerable variation in 
non-technology based learning, the most common and traditional model can be characterised 
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in the following way. 

Resource Negotiation: there is usually no real negotiation - every learner gets the same 
resources (lectures, reading lists etc), although it may be possible to get some individual 
attention if the course group is not too large. Likewise the learner's responsibility is simply to 
return assignments or essays, or to pass examinations. 

Coordination: in the most traditional of approaches, coordination simply does not exist - 
students are not expected to work together in any way, and if they do, no provision is made 
for this. Those teachers that do support group learning usually structure this within existing 
lessons, and no provision is made for group assignments or other actions outside the 
timetabled sessions. So the only coordination that applies across all formal teaching is 
achieved through timetabling, which ensures that all students get exactly the same diet of 
teaching and laboratory activities as everyone else. 

Monitoring : this may be achieved through forms of continuous assessment, or through one to 
one tutorials, where teachers can examine how a student is developing their understanding 
and competencies. However, when class sizes are large, it is usually this aspect that suffers 
most. In some cases, teachers can amplify their monitoring capability by using post graduates 
as group tutors; in other cases personal tutors can play this role, although if the subject area 
is too far from their own expertise this can be less than satisfactory. 

Autonomous learning: Students have always been able to use libraries to find additional 
materials to support their learning. This is perhaps what has ameliorated the lack of individual 
support offered by courses. However guidance on how best to find what they need has 
always been in short supply. 

Self organisation: there is little evidence of any formal structures helping students to work 
organise themselves into work or study groups. While students may form study groups, 
neither time, space nor advice is provided for this, except in a few examples of study skills 
courses advising on group learning. 

Adaptability: teachers can adapt their teaching to the needs of their learners, by changing the 
content of their lectures, the way they present them, and the supporting resources they use. 
However, these approaches tend to be coarse grained, responses to the perceived needs of 
the whole group, not individuals. 

It is this last point that is the most significant. The formal set up implies that students following 
a course of study are to a large extent similar, and so an identical teaching approach will 
meet all their needs. Indeed, the process of selection also implies that the learner group is 
homogeneous. Formal teaching is designed to allow the teacher to treat the learner group as 
if they were one. However, the fact that the class does not consist of a single student 
constrains one key aspect - the ability of learners to engage in one to one conversations with 
the teacher. In a formal situation the conversational demands of a class of students would 
ovenwhelm the teacher, whereas if they had only one student, the balance between 
monologue (lecture) and dialogue would most likely be very different. 

All these issues - resource negotiation, coordination, monitoring, adaptability, self 
organization, and individualisation can be affected by the use of technology. How they are 
affected depends on the facilities and flexibility particular systems offer. We believe that, in 
addition to the conversational criteria outlined earlier, these organisational criteria are crucial 
when considering the choice of VLEs. 

3.3 Constructing an Evaluation Framework using the VSM 

The methodology we shall use for evaluating the VLEs against the model just described is to 
outline what tools each provides for accomplishing the six tasks set out in the model. To 
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illustrate this process we provided three examples of this approach using respectively: 
CoMentor, Asymetrix Librarian and Learning Landscapes. 

VLE1: CoMentor 



VSM Functionality 


Support within VLE 


Resource Negotiation 


The learning contract using CoMentor can be as loosely 
or tightly structured as a tutor wishes. A resource library 
is available within the system as are places for students 
to do their own or group work. CoMentor is designed to 
be used in institutions to supplement existing courses, 
rather than to provide a whole on-line course. CoMentor 
could be used to open the negotiation of a learning 
contract for the continuation of a course or to elicit 
understanding of a particular topic 


Coordination 


Collaborative working is a key feature of Co-Mentor's 
design. Threaded discussion lists with sub-group 
membership allow students control over with whom they 
share their work. A shared concept mapping tool is also 
provided 


Monitoring 


Tutors can monitor the discussions in CoMentor, taking 
part if appropriate. 


Individualisation 


There is a repository of on-line resources within 
CoMentor which students are encourage to use to find 
their own materials. Additionally they have access to 
previous students work. This is a novel feature within a 
VLE and is potentially an extremely useful tool in 
helping to guide students thinking and to build on work 
that has already been accomplished. There is a list of 
people who can provide expert advice on selected 
topics. 


Self-Organisation 


Students can organise their own work groups in 
CoMentor from which the tutor can excluded, if that is 
desired. In this sense CoMentor is a tool that belongs to 
the students as much as it does the teaching staff. 
Students can send e-mail to each other and tutors 
within the tool and there is a chat facility for 
synchronous conversations. 


Adaptation 


The inherently dynamic nature of the CoMentor 
environment means that adaptation is not an issue in 
the same way as it is with systems designed to deliver 
courses. The content of CoMentor is encapsulated in 
the discussions, work and resources it contains and as 
such it is a continuously adapting structure. 



VLE2: Librarian 
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VSM Functionality 


Support within VLE 


Resource Negotiation 


There is the capability for resource negotiation in 
Librarian. Students can self enroll in courses or modules 
which tutors have made available. The availability may 
be determined by previous conversations with the tutor 
or fulfilling conditions such as prerequisites. The student 
can browse available courses and also search the 
database for collaborations or lessons that meet specific 
criteria. Tutors can assign learning activities at various 
levels of granularity to individuals or groups 


Coordination 


Administrators can set up collaborations to encourage 
students to work together. The software includes built-in 
threaded discussion lists but can also accommodate 
third-party collaboration software. 


Monitoring 


Librarian has a powerful reporting facility that allows a 
tutor to monitor various aspects of a students 
performance and progress, including item analysis, 
content usage, organization membership and lesson 
score reports. 


Individualisation 


Students can search the Librarian database for 
resources or other materials and of course the WWW. 
There is not an area for students to present their work 
or contributed materials to each other, although they 
can contribute materials as attachments to e-mails or 
discussion postings. 


Self-Organisation 


Students cannot set up their own collaborations or 
construct their own learning activities in Librarian. 
Student interaction and organisation outside of the 
tutors influence is limited to individual e-mails 


Adaptation 


The Librarian administrators interface supports easy 
assembly of an organisational structure using a 
hierarchical tree. Lessons can be organised into nested 
blocks and modules using this interface. This allows 
learning activities to be pulled out, modified and 
replaced at any point during a course. Because people 
are also represented in this structure, individual 
assignment of modified activities can easily be 
accomplished. 



VLE3: Learning Landscapes 



2Er|c 



29 



6/14/00 8:50 AM 



A Framework for Pedagogical Evaluation of Virtual Learning Environments 



http://www.jtap.ac.uk/reports/htm/jtap-041.html 



VSM Functionality 


Support within VLE 


Resource Negotiation 


In Learning Landscapes (LL) people can be assigned to 
learning activities or learning activities can be assigned 
to people (either individuals or sub-groups), thus the 
system can support either a traditional or 
resource-based approach. Conversations are central to 
the design of LL, so that as soon as a person is 
assigned to an activity, a conversation thread is started 
in the person's space within the activity. Further 
specialised resources or sub-activities can be 
negotiated from this conversation thread. 


Coordination 


All members of an activity group or sub-group have 
access to a discussion forum accessible from the 
activity space. This organisation within the software 
naturally contextualises the discussions to the 
appropriate people and level. Students can also mail 
other group members individually in the same way. 


Monitoring 


Tutors can monitor discussion threads and their own 
conversational histories with individuals. They can also 
check whether assignments have been returned and 
grades or view students' homepages for further 
information on individuals. More detailed progress 
tracking tools would be useful. 


Individualisation 


The system includes a web browser in it. Students can 
use it to turn interesting on-line pages into resources in 
their personal resource structure. These can be posted 
to selected group members as sub-learning activities 
(there is no difference between the tutor's version and 
the student version). 


Self-Organisation 


As mentioned above, students have the ability to 
construct and host learning activities without recourse to 
the tutor, thus a variety of special interest or project 
groups could be spawned within a learning activity 
group. This highly interactive approach requires the 
student to participate responsibly. 


Adaptation 


LL is specifically designed for easy adaptation such that 
the initial course or overall learning activity sent out by 
the tutor is adapted and added to continuously as the 
course progresses. 



3.4 Summary 

In this section we have introduced the Viable Systems Model (VSM) and shown how the 
model can be adapted for use in an educational context (Liber, 1997). We then proposed that 
core processes within the adapted model could be used as criteria against which to evaluate 
the functionality of VLEs and also to gain a better understanding of the educational model or 
models to which they are best suited. One of the main benefits of the VSM is that it probes a 
whether a system can support a resource-based, student-centred teaching approach. 
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amplifying the teacher's variety and attenuating the variety in the learner group to make the 
teaching approach viable - BUT without treating all learners as if they were all the same. We 
illustrated the VSM framework for evaluating VLEs using three examples drawn from the 
systems reviewed in section 2 - Librarian, CoMentor and Learning Landscapes. 

One of the most important factors here is that students should have the capacity to organise 
themselves so that all the study is not solely tutor led. In terms of virtual learning environment 
this means that learners should be able to set up their own study groups and collaborations. 
Quite a few systems do not allow learners the permissions to do this. There should also be 
quite detailed information about different students interests available within the VLE. Thus 
students can locate others within their groups with whom they share common interests or 
goals. A good way to do this by leveraging web-technology is to have a link to each person's 
homepage as well as their e-mail. 

Another important feature that emerges from this analysis is that students should be able to 
contribute their own resources and materials to the group (monitored by the tutor). Again this 
means that the responsibility for the shaping the learning content shifts from the tutor to the 
students. The tutor's time is then freed up to monitor discussions and engage in conversations 
or resource negotiation with individuals or sub-groups. 

The third key feature of a VLE from this perspective is that it should be easy to continuously 
adapt and individualise the structures, content and activities it contains as the course 
progresses. It should also be possible to archive completed learning activities for future 
re-use. 

4. Discussion and Conclusions 

4.1 Discussion 

At the start of this report we identified a number of ways in which virtual learning environments 
may assist with learning and teaching in higher education. Most of the benefits of VLEs lie in 
their potential to support styles of learning that are especially time-intensive for university 
teachers using traditional methods, but which have always formed a core part of a university 
education. We are referring, in particular, to: 

o collaborative learning 
o discussion-led learning 
o student-centred learning 
o resource-based learning 

Typically these learning styles have been supported using tutorials, seminars and small-group 
project work. However these activities have increasingly been squeezed out of undergraduate 
timetables in recent years due to factors such as increased time pressures on staff, increased 
student numbers and increased economic pressures. 

As the admissions net has been widened, so too has the diversity of students - in terms of 
their backgrounds, age, experience, education and expectations. Yet due to the pressure of 
larger intakes, teachers have been forced to sacrifice precisely those activities that allow 
diversity to be managed positively, in favour of a uniform content delivery model that has 
economies of scale, but which essentially ignores the variety of the students. 

As Laurillard (1993) states, "it is not that teachers want to teach this way any more than 
students want to struggle to learn in this manner, both parties are constrained to operate 
within a university system over which they have limited control and which is barely capable of 
withstanding external the pressures currently being exerted upon it". 
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The hope, expressed by university teachers and administrators then is that networked 
computer systems such as the VLEs we have reviewed here will allow more flexible teaching 
and learning styles to be adopted whilst at the same time reducing the time burden on the 
teacher. 

We investigated a number of well-known or innovative VLE products (details may be found in 
appendix 1), some of which are commercially produced, others are the result of university 
development projects. For each VLE we compiled a feature set list from the various sources 
available to us (see appendix 2). We argued that although feature sets give an indication of 
the individual tasks a software package can perform they fail to capture the overall picture of 
how well designed the software is for supporting the integrated student activities listed above. 

In response we put forward two models, one from an educational perspective the other from 
an organisational perspective that might be used to provide a more effective evaluation 
framework for VLEs. The two models are complementary in many ways, having different 
scope and orientation. 

The conversation model focuses on interactions between an individual student and tutor. An 
evaluation from this perspective helps to identify whether a VLE is set up to allow 
individualised activities to be constructed for a student. The activity should be based on a prior 
discussion with the student that has identified any mismatch in the conceptual domain 
between tutor and student and the VLE should support this process. 

The VSM model focuses on how the software helps a tutor manage conversations and the 
construction of individualised activities for a large number of students. In order for a VLE to 
effectively support the tutor in doing this, it must not only be easy to adapt on the fly but also 
provide for student self-organisation, resource-gathering and publication of material to the 
system. 

In summary, used in conjunction, these two models provide an evaluation framework for 
VLEs. There is considerable scope for further elaboration of the scheme we have 
implemented here. 

One of the problems with evaluating VLE packages that are currently available is that, with 
some exceptions, very few are currently being used for real within institutions. As a result, 
data and feedback from users is hard to collect. We expect this situation to change rapidly in 
the near future. 

4.2 Conclusions 

On-line learning technologies and virtual learning environments in particular are attracting 
considerable attention with in UK higher education and it is likely that emerging systems will 
have considerable impact on the way that learning and teaching is conducted in universities. 
Widespread uptake has yet to become apparent however, and the reasons for this are not 
entirely clear. Part of the problem that has been the primary focus of this report is that this 
field is still in an early stage of development and an appropriate framework for evaluation of 
different systems has not yet been established. 

Perhaps the most important conclusion to be drawn from this report is that we are not 
providing an evaluation strategy that helps discriminate VLEs based on some notion of 
quality. Instead we have produced a framework that can help to reveal the underlying 
pedagogical assumptions and orientation of the software, thus providing teachers with a basis 
upon which to choose a VLE according to how they want to teach. 

This point brings us back to the issue of whether choosing a VLE is an institutional-level 
decision or a responsibility that should be left in the hands of individual teachers. It raises the 
question of whether it is possible (or indeed desirable) to define teaching strategy at an 
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institutional rather than individual level. 

Higher education is changing. The student body is becoming less uniform and knowledge is 
growing and changing faster than ever before. Industry's demands are also more volatile, as 
they struggle to become more competitive in an increasingly globalised economy. Their 
employees need up to date knowledge and skills, and updating these requires training and 
education at the time and place they need it. While traditional courses will continue to be a 
major part of universities provision, the most likely growth area will be in 'lifelong learning' 
courses offered to industry or to individuals. The virtual learning environments of today and 
their successors are likely to form key strategical aspects of teaching and learning in 
universities of the future. The work contained in this report represents a first step towards 
recognising the educational and organisational requirements of this key learning technology. 
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Appendix 1 

In the sections below we describe the features, technology and scenarios of use of a number 
of VLEs that are currently available. The information in the following descriptions is drawn 
from documents produced by the vendors / creators of the software, referenced reviews and 
evaluations and personal experience. 

1. Learning Space 

Overview 

Learning Space is based on Lotus Notes and uses Notes Server technology to provide a 
secure environment with a rich set of tools. The basic architecture of Learning Space consists 
of five main databases that are used to manage different aspects of course development and 
use: 
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1. The Schedule - provides an individualised schedule manager 

2. The Media Center - acts as a repository for multimedia course materials 

3. The CourseRoom - is a conferencing tool that allows the creation of threaded discussions 
for student interaction and group work on assignments 

4. The Profile Manager -contains basic information on participants similar to a homepage or 
online CV. 

5. The Assessment Manager - is a purely tutor's tool for privately testing, marking and giving 
feedback on participants work. It also allows a gradebook view of all participants on a 
course. 

Learning space includes tools for browsing the web and inserting multimedia material into 
learning space documents. Links can be defined from Learning Space to multimedia content 
on the web. Asynchronous communications tools are based around e-mail, which is used for 
private one-to-one discussions and the CourseRoom where threaded private or public 
discussions can be facilitated by a tutor. Synchronous communications are supported by a 
variety of tools available with Learning Server 2.0 such as chat, whiteboard, video and 
teleconferencing. Additionally resources and other content may be exchanged via the Media 
Center. Students can maintain a progress-tracking folder containing a record of their own 
work and assignments throughout the course. Completed courses may be archived by the 
instructor for future use. A Portfolio is contained in every participant's Profile. This is a secure 
area for returned assignments and assessments which can only be viewed by the participant 
and the tutor. 

Technology 

Client-Server architecture. The Learning Server will reside on a Windows NT or Unix Platform. 
The client is cross-platform requiring either a notes-client (required for the tutor) or a web 
browser capable of supporting frames and an internet or LAN connection. 

2. WebCT 

Overview 

According to the team that developed WebCT it is " a tool that facilitates the creation of 
sophisticated Web-based educational environments." (Goldberg and Salari, 1997). It can be 
used flexibly to create entire online courses, or to publish materials that supplement existing 
courses. All interaction with WebCT takes place through a web browser.. Essentially a webCT 
course consists of a series of linked HTML pages that define a path or "road-map" through the 
course material. The course content is supplemented by webCT tools which can be built into 
the course design by simply dragging the appropriate tool icon onto the web page. This 
creates an active link to a 'toolpage'. 

There are three main aspects to webCT: 

1. A presentation tool that allows the designer to customise the look and feel of the course 
pages. 

2. A set of student tools that can optionally be integrated anywhere in the course. 

3. A set of administrative tools to aid the tutor manage the course when it is in progress. 

A webCT course centres around a single course homepage, which appears whenever a 
student logs on to their course. This page contains links to the various course pages and the 
tools. The learner tools in webCT include: asynchronous communications tools (e-mail and 
conferencing) and also a chat facility; student evaluation and self-evaluation tools such as 
online quizzes and MCQs; an image repository; a glossary database; learner collaboration 
and presentation areas; content annotation, homepage generation, course navigation and 
searching tools. Tutors can track student progress including the number of times and when 
they have accessed the course. 
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Technology 

This is a client-server system with all interaction through a web browser client. Supported 
server platforms are currently only Unix but an NT 4.0 version is to be announced. 

3. TopCIass 

Overview 

TopCIass courses are constructed of Units of Learning Material (ULMs). These ULMs can 
consist of pages, exercises, or further ULMs themselves. ULMs can be freely exported and 
imported from course to course. In addition to course management, TopCIass also manages 
student progress, tracking, and access to course materials. 

The learner tools in TopCIass include web browsing and embedding of hyperlinks into 
documents. Asynchronous communication and sharing is supported through e-mail, threaded 
discussion groups and BBS file exchange. Synchronous tools are not supported. There is a 
self testing facility and progress on coursework can be tracked by both tutor and student alike. 
It is also possible to produce individualised workplans for students. 

The tutor tools in TopCIass appear to have been well designed for ease of course 
construction and adaptation of courses in progress. ULMs are nestable learning units which 
are built into a course using a hierarchical outlining tool. Students are divided into groups 
according to a class model and access to discussion forums can be restricted to the class. 
Also ULMs can be easily created and assigned to particular subsets of students. 

Technology 

TopCIass server software, web browser as client, LAN or Internet connection 

4. Virtual University 

Overview 

Virtual-U like the other systems described is a server based integrated environment for 
education that uses a web client. One of the major differences between Virtual-U and other 
systems is the use of a campus metaphor within which locations and objects are used to 
represent the different tools and activities. The homepage contains a map of the Virtual_U 
campus which consists of : 

o A Course Room - containing details of the courses 

o A Library - containing resources and links to library pages and search engines 
o A Gallery - for presentation of multimedia resources 
o A Conference Building - from which conferences can be accessed 
o A Cafe - another conference space for more casual student or staff interaction 
o A Personal Workspace - The workspace provides links to all other areas of the campus 
and also includes a number of tools for managing your own work. These tools include a 
calendar for personal time planning, a submissions box for viewing assignments, a 
grade-book allowing students to view their own performance or for instructors to monitor 
student progress, a glossary and preference options. 

Additionally, there is also a detailed help system providing course designers with pedagogical 
guidelines for the use of the system. 

Technology 
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Web browser client, Server software runs on Unix or NT. 

5. Web Course in a Box 

Overview 

Web Course in a Box (WCB) is one of the earliest integrated VLEs to have been commercially 
available. All access to a WCB course is through a web browser. It presents a customisable 
course homepage on log in which contains links to the various features of WCB. These 
consist of: 

1. A course syllabus providing an overview of the course, prerequisites, instructor etc. 

2. An announcements board. 

3. A course schedule. A tabular presentation gives dates and hyperlinks to the course units 
and materials themselves 

4. Student Information. This includes the students' homepage and e-mail contact. 

5. Learning Links. This where discussion forums, shared whiteboard and links to learning 
resources reside. 

6. Help / Utilities. This link takes the user to a panel for editing personal material and 
preferences and also provides a help system. 

Technology 

WCB server software and course authoring tools. Web browser client. 

6. Course! nfo 

Overview 

Courseinfo is another example of a robust commercial VLE package which is based around a 
client-server architecture where the server software is accessed through a web client. It 
incorporates many features in common with systems described above which are accessible 
from a course homepage that is created and customised by the course instructor. Features 
include: An announcements board, course pages, course and staff information, an 
assignments space, communications tools, student tools and search tools. The 
communications tools include both asynchronous discussion facilities and chat and 
whiteboard functionality. Student tools include a calendar, homepage editing, a 
grade-checking function and a drop-box for sending assignments. There are also self-testing 
tools. 

Technology 

Blackboard Server for Unix or NT, web client. 

7. FirstClass Collaborative Classroom 

Overview 

FirstClass is slightly different to many of the systems mentioned here in that it is primarily a 
conferencing system with robust e-mail and collaboration tools. But with the introduction of the 
First Class Intranet Server (FCIS) it also allows web publishing of course materials, personal 
websites, and rapid application development of applications to run either over the web or in 
the FirstClass client. First Class is probably the most widely used conferencing system in UK 
HE currently. A notable example is the OU who support over 35,000 users with FirstClass. 
Important features of FirstClass from a student perspective are the ability to work offline. 
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search facilities, a 'chat' tool. From a teacher or administrators viewpoint the ability to publish 
web content easily means that on-line courses can be offered with FirstClass and interactive 
simulations can be built using the RAD tools. In addition the client Ul is highly customisable. 

Technology 

FCIS server runs on MacOS or Windows NT and a FirstClass or web client required 

8. Librarian 

Overview 

Librarian provides complete VLB functionality. Administrators have a well-designed interface 
for administration and course construction activities. A hierarchical tree model allows 
'organisations' to be defined and placed in the tree. An organisation consists of members, 
learning activities and sub-organisations. This allows any organisational structure to be 
modelled in the tree. Learning activities may be similarly organised into blocks and modules, 
where each block may contain one or more modules and a module directly contains some 
learning content. Learning activities may consist of a combination of course materials with 
assignments and on-line discussions for collaborative work. Students may sign up for any 
block or module of interest, although pre-requisites on access may be set by the course 
administrator. Librarian also provides progress tracking and search facilities. 

Technology 

Librarian is a Java-based learning management system that runs on a Windows NT or UNIX 
platform. Its client/server architecture enables learning delivery across the Internet or intranets 
and is designed to scale with the growth of an organization. 

9. ARIADNE 

Overview 

ARIADNE is a the result of a European funded project form a consortium of European 
universities with its primary aim to enhance sharing and re-use of computer-based teaching 
material. In the words of the project leaders "ARIADNE explicitly addresses four aspects of 
the teaching process: producing computer-based teaching material, managing this material to 
permit sharing and re-use, assembling courses, and delivering courses to students. 
...ARIADNE has developed tools for creating simulations, multiple choice questionnaires, 
self-assessment exercises and tools for segmenting text and video. Hypertext generation is 
also supported with the aid of a text conceptual segmenting tool." 

Technology 

ARIADNE server, web client (Java enabled) 

10. CoMentor 

Overview 

CoMentor is a web-based software system, developed at the university of Huddersfield, that is 
primarily focussed on facilitating discussions and collaborative work on-line. At the heart of the 
system is a MOO architecture implemented on a web server with spaces in which students 
can participate in real-time or asynchronous discussion. In addition there are learning tools to 
support and structure debates and collaboration. In particular students can work together over 
examples from archives of previous students' work. The system includes an individual 
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work-space, a group work-space, and a resource archive. There is also an announcements 
board. CoMentor is not intended to provide complete on-line courses but rather to enhance 
existing courses. 

Technology 

CoMentor runs on a Unix platform and can be accessed from any Java-enabled web client. 

11. CoSE 

Overview 

COSE is a VLE that arose from a project at the University of Staffordshire to build tools for the 
creation of study environments rather than delivery of materials. The software was designed 
to support a constructivist pedagogy based on principles of good practice. The idea behind 
COSE is that a course is a group of people to whom learning opportunities can be flexibly 
assigned, rather than the more traditional view that it is a body of content to which people are 
assigned. Cose has built-in asynchronous discussion tools and work-sharing tools. COSE 
provides a publication mechanism for content that includes metadata and is one of the few 
systems described here to take the issue of metadata seriously. Tutors can easily assign and 
de-assign learning opportunities to groups, sub-groups and group managers can be created 
and all subgroups have their own noticeboard. Learners can create content as 'COSE' pages 
and optionally attach other sorts of files to them. Thus submission and return of assignments 
is made simple. 

Technology 

Like the other systems COSE is platform independent using a web client. The COSE server 
software should run on any server platform running Apache 1.0+ and Perl 5.003+. It has been 
tested on NT4, Linux 2.0, and DEC Unix. 

12. Learning Landscapes 

Overview 

Learning Landscapes is a Java software application developed at the University of Wales - 
Bangor that supports on-line interaction between student and tutor to allow the negotiation, 
creation and management of study programmes. One key difference between Learning 
Landscapes and other systems described above is that is a distributed rather than client 
server application, this means that no server management is required and that offline working 
is feasible. The software is specifically designed to handle the management of 
resource-based collaborative learning. With Learning Landscapes you can: 

o develop and maintain detailed information on students 
o develop and maintain detailed information on learning resources 
o browse web based resources using the in-built browser or an external browser 
o put resources and students together to create courses (or modules, or units), 
o structure courses into sub-learning activities to any depth 

o automatically send course structures and learning resource references to students 
o send and receive assignments and assessments 
o dynamically update these if you add resources or students 

o have email based discussions with groups and individuals that is structured according to, 
and accessed from within the course 
o store these conversations as part of students' learning profiles 

Technology 



3Er|c 



38 



6/14/00 8:50 AM 



A Framework for Pedagogical Evaluation of Virtual Learning Environments 



http://www.jtap.ac.uk/reports/htm/jtap-04 1 .html 



Java 1.2 application, no server required. 
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Virtual Learning Environments Questionnaire 

Part 1 . Respondents' Details 

Notes; This section may be left blank if you wish to remain anonymous. 

Name of Institution 

Respondent's Position 

Contact e-mail; 

Part 2. About teaching and learning at your institution 

Approximately how many students are enrolled on degree courses at your institution? 

What proportion of your students are enrolled on part-time courses? (approx.) 

What proportion of your students are distance learners? (approx.) 

Part 3. Description of systems currently in use 

Notes; In this section we are trying to gather information about what systems (if any) are currently used at your 
institution 

Do you currently use an integrated Virtual Learning Environment for teaching and learning at your institution? 
YES / NO 

If YES, please mark as appropriate; 

WebCT TopCIass LearningSpace Virtual-U 
Courseinfo Web-in-a-Box Forums FirstClass Classrooms 
Other (please specify) 



Do you use a conferencing system or other groupware product for teaching and learning purposes? YES / NO 
If YES, please mark as appropriate; 
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Domino / Notes FirstClass MS Exchange Collabra / Newsgroups 
Other (please specify) 



Do you use the WWW for teaching and learning? YES / NO 
If YES, please mark as appropriate: 

Lecture Notes 

Other resources (Please describe below) 
Other activities (Please describe below) 



Do you use videoconferencing or other synchronous communications system to support group working, either at 
a distance or with computer-based resources? YES / NO 

If YES, please mark which technologies you use: 

Videoconferencing Whiteboard Shared Browsing Chat 

Please also mark which (if any) of the following products you use: 

CU SEE ME NetMeeting Timbuktu Netscape Conference 

Internet Relay Chat 

Other (please specify) 



Do you use net-based virtual laboratories, interactive simulations or other interactive courseware? YES / NO 
If YES, Please supply details 



Do you use any other system or systems that you think we may be interested in? YES / NO 
If YES, Please supply details 



If you have answered no to all of the above questions, please return the form to us at the address at the top - this 
information is still valuable! If you have responded affirmatively to any of the above questions please fill in the 
next section to the best of your ability. 

4. The properties of virtual learning environments 

Notes: In this section we are aiming to query your impressions of the system that you are currently using in order 
to gain a rough idea of the capability of various systems. Unless you are using an integrated virtual learning 
environment not all of the subsections will be relevant. These sections use a Likert scale rating scheme based on 
the suitability of the tool for performing various tasks. The scale key is as follows: 
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1 = not at all 

2 = poor 

3 = satisfactory 

4 = good 

5 = excellent 

For each question please circle the appropriate number. 

4.1 structuring of courses, resources and learning content 

Does your system provide a facility for tutors to construct courses by embedding content, task descriptions, 
on-line resources into a structured course outline? YES/NO 

If Yes, please rate your system on the following criteria. 

How suitable is it for aggregating learning units and / or content? 

1 2 345 

How adaptable are the structures once they have been created and are in use? 

1 2345 

Can a structured problem space be easily created for students to work in? 

1 2345 

How easy is it to support individual learners’ needs? 

1 2345 

How specifically can informational resources or references to resource locations be incorporated into projects? 
(e.g. hypermedia support) 

1 2 345 

Is the system suitable for incorporating a wide range of multi-media resources and content? 
1 2345 

How easily can non-proprietary software components (e.g. JavaBeans / ActiveX components) be embedded in 
the tool 

1 2345 

4.2 Handling Workflow 

A fully integrated VLE should be able to handle all the workflow requirements between students and tutors in a 
flexible and context-sensitive way. It should also be able to handle a range of pedagogical models from a 
traditional delivery model to collaborative approaches. 

Does the tool provide shared workspaces? If so please rate the quality 

1 2 345 

How well does the VLE provide for negotiation and development of individual learning plans? 
1 2 345 

How easy is it for tutor and student to collaborate in constructing a problem domain within the virtual 
environment? 

1 2 345 
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How well does the system support collaborative working of a number of students on the same project? 
1 2345 

Does the system support submission of assignments from student to tutor? If so please rate the quality 
1 2 345 

Does the system support recording and return of assessments to students? How well does this facility meet your 
needs? 

1 2345 

How well does the system accommodate off-line as well as on-line working? 

1 2345 

4.3 Communication Tools 

If the system uses e-mail to support one-to-one conversations, how easy is it to use in a teaching / learning 
context? 

1 2345 

If conferencing software is integrated with the system in order to support group discussions and group working, 
how satisfactory is it? 

1 2345 

If the system uses any form of synchronous communications, how useful are these components? 
1 2345 

4.4 Student Profiling / Progress Tracking Tools , 

How well does the tool allow tutors to track students' progress? 

1 2 345 

How rich a picture of an individual student's background, interests and aspirations does the tool provide? 
1 2345 

4.5 Integration Issues 

How well integrated are the various tools provided by the system? 

1 2345 

How easily can content from external sources be integrated into the environment? 

1 2345 

Is the system IMS compatible? 

1 2345 

How well does the system inter-operate with administration systems (e.g. student records databases etc.)? 
1 2345 

4.6 Functional / Technical Issues 

How easy is it to set up and maintain the system? 

1 2345 
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How easy is it to learn to use the system for tutors and students? 
1 2 345 

How time-consuming is it to enter data into the system? 

1 2 345 
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